The Principal Components Method as aPre - processing Stage for Decision Tree

In this paper we use the principal components method as a pre-processing stage for decision tree learners. We investigate the case when query examples are known in advance. We have evaluated an easily computable criterion for choosing the optimal number of principal components. Our experimental results show that decrease in error rate can be achieved without an increase in hypothesis complexity (the size of a decision tree). On some datasets the method competes even with C5.0 with boosting. We also show that the required time does not exceed sig-niicantly the time needed for learning a decision tree without the use of principal components. We also describe experiments with other proposi-tional learners. Modest gains have been observed for the more common case when the query examples are not known in advance. 1 Motivation A serious drawback of decision tree learners like C4.5 12] is its very restricted language. In the process of knowledge discovery in databases (KDD) it is very often the pre-processing stage, namely transformation and reduction of data, that play an important role. A good transformation/reduction technique may result in new attributes that are more appropriate for a data mining algorithm. A wide range of constructive induction techniques 6, 7, 9, 15] has been explored aiming at enriching the language of propositional learners 4, 5]. The usual way is to add new attributes that are computed from existing ones using a priori given functions. One important class of functions includes linear combinations of numerical attributes. To prevent a large increase in computational complexity and to prevent over-tting we cannot add just any linear combination. Principal components 11, 13] are useful linear functions that capture linear dependencies among attributes and hence deserve particular attention. The method presented can be seen as a multistrategy approach 5] consisting of two steps. In the rst step we compute the principal components from numerical attributes. In the second step we use these principal components to replace, or to add them to the original attributes. The rst step is actually a form of constructive induction 4] applicable for numerical variables, which uses the methodology